Devil in the Details: Assessing Automated Confidentiality Classifiers in the Context of NATO Documents

Automating security classification of documents has a great potential to increase the efficiency of information management and security in IT systems used by governmental, military and international organizations. In particular, automated security classification can be used in support of cross-domain information exchange solutions, such as the NATO Information Clearing House. These solutions often require a manual review of documents flowing between different security domains and thus introduce a performance bottleneck. In this paper, we describe an automated confidentiality classification process that could offer an important support for the manual review of documents. It consists of providing an automated pre-labeling of documents, accompanied by an assessment of confidence levels concerning the identified labels. This would allow responsible personnel to focus on low-confidence cases and review other documents only to the extent required to provide an appropriate audit and security control. We evaluate performance of some of the freely available classification algorithms in the context of confidentiality classification of NATO documents and conclude that although these systems are not accurate enough to warrant a complete autonomous operations, they are effective enough to provide an important support for human operators.